Skip to content

[SYSTEMDS-3971] Add Scuro evaluation layer tests - #2639

Open
shieru1214 wants to merge 1 commit into
apache:mainfrom
shieru1214:scuro-eval-tests
Open

shieru1214 wants to merge 1 commit into
apache:mainfrom
shieru1214:scuro-eval-tests

Conversation

@shieru1214

@shieru1214 shieru1214 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

This change adds tests for the Scuro evaluation layer in two new files. test_split_aware_evaluator.py checks split-aware evaluation, and test_task.py checks cross-validation in Task.

Split-Aware Evaluation

  • EvaluationSplit: The test checks three invalid splits: repeated fit IDs, repeated evaluation IDs, and an ID shared by both parts. Each case should raise a ValueError.
  • Supervised fit: A supervised TfIdf subclass records the inputs to each fit call. The test checks that the instance IDs and labels come only from the fit part of that fold.
  • Node cache: The test builds a TfIdf → MLPAveraging DAG and checks that the node cache stays empty when fitted results are not shared by default. A fitted result depends on the current fold, so it should not be reused across folds by default.
  • _leaf_input: The tests check that a leaf selects the right input by modality ID and representation index, including when the modality ID is a string. An unknown modality or an index out of range should raise an error.
  • _align_partition_shapes: The tests check that shorter partitions are padded with zeros to the longest length, while the longest partition stays unchanged. Data should also stay unchanged when the shapes are already the same or the ranks are different.
  • materialize: The test checks that a modality with neither data nor a loader raises an error.

Cross-Validation in Task

  • Grouped folds: The test uses 20 instances in 10 groups. It checks that the same group does not appear in both the fit and evaluation parts of a fold.
  • Train and test groups: The tests check that a group shared by train and test raises an error. When overlap is explicitly allowed, the task records the shared group.
  • Stratification: The test checks that stratification is enabled for single-label targets but not for multi-label targets. Neither case has a fallback reason. It checks the stratification setting, not the label balance in each fold.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant